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DATA ANALYSIS OF THE 



TEACHING AWARD BALLOTS 
by 



Robert R. Read 



I. INTRODUCTION 

The annual award for excellence in teaching was established 
by the Superintendent in 1969. The first recipient was honored 
at the Spring graduation in June 1970. There is a cash prize 
(which has become substantial due to an endowment by Rear Admiral 
John J. Schieffelin) and, in recent years, the committee has 
provided the Provost with a list of honorable mention faculty 
who, as a result, have received step increments in pay. Further, 
the winning instructor's name is etched on a plaque in the library. 

The recipient of the award is chosen by a committee 
of faculty which conducts a secret poll of on board students 
and recent former students. The data produced by the poll 
are machine processed. The output is coded so that the numer- 
ical summaries cannot be associated with the names of the 
particular faculty involved, and the winner is selected in a 
totally objective fashion. The membership of the committee 
rotates. Each new committee has exercised its responsibilities 
in terms of defining the set of faculty eligible to receive the 
award and the set of franchised voters to participate in the 
poll. Also, the specific use made of the numerical summaries 
may change from committee to committee. Major changes in the 
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structure of the ballot have been avoided since such changes 
would entail substantial changes in the computer codes. 

The basic format of the ballot was set by the initial 
committee as was the nature of the data summaries ^and the author 
played a substantial role in these activities. Over the years, 
the uses made of the balloting data have increased (i.e. an 
honorable mention list is extracted and previous year's per- 
formance data is used in selecting the winner) and the time 
came for revising the data processing effort in the light of 
this. Because of this need the author was placed again on 
the selection committee (1978) and, after study, recommended 
revisions in processing, monitoring, and record keeping. The 
purpose of the present report is to document the support for 
and nature of the changes. 

The basic recommendations are fourfold: 
i) Modify the basic scoring system so that 

(a) the correlation between high scoring instructors 
and instructors who are identified on few ballots 
is removed; 

(b) the practice of listing large numbers of comparable 
faculty on a ballot in an effort to boost the 
prime candidate is discouraged. 

ii) Prepare paired comparison studies in parallel to the 

basic scoring system so that attention is drawn to those 
instructors whose basic scores are relatively low because 
the set of instructors with whom they were compared was 
an unusually strong set. 
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iii) Organize the record keeping process so that there is 
a one-to-one correspondence between instructors and 
their identification numbers. Maintain historical 
records of each instructor's performance in previous 
years' balloting coded by A, B, E, I, representing, 
respectively, top 5%, next 15%, eligible but not in 
top 20%, ineligible. 

iv) Monitor the data each year so that the goals in i) and 
ii) are maintained. Perform ballot population studies 
so that the selection committee will be made aware of 
anomalies in voting patterns. 

The organization of the report is as follows. Section II 
contains some historical information about the structure of 
the ballot, control of errors, and choice of notation. In 
Section III is described the early approach to summarizing 
the data, scoring, and choosing the winner. The pertinent 
experiences and biases observed in the data processing are 
described in Section IV. A paired comparison study of the 
data is given in Section V. This leads to Section VI which 
summarizes the results and describes the newly implemented 
system. Four appendices are included which contain many of 
the supporting details and data analyses. 
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II. BACKGROUND AND NOTATION 

The first Selection Committee (chaired by Prof. T. Gawain) 
struggled with the issues of what data to collect and how to 
collect it. It decided to gather information, by secret ballot, 
largely from the consumers of instruction. Originally, the set 
of franchised voters consisted of students on board, faculty, 
curricula officers, and recent alumni. Each voter was asked to 
specify the population of instructors (from a list of eligible 
instructors) that they felt competent to compare, and from 

this population they were asked to name their first, second, 
and third choices. The ballot had to list at least five 
instructors in order to qualify for inclusion in the system. 

Also some biographies were collected such as voter category 
(student, alumni, faculty, curricular officer) and curricular 
area (aeronautical engineering, operations research, etc.). 

Each ballot contains space for voluntary written remarks 
supporting its first choice. 

The ballots are reviewed for general validity by a 
neutral source and the data are transferred to IBM punched 
cards. Each ballot is given an index number so that it may 
be retrieved if necessary. The written comments are coded 
for the computer only in terms of their presence or absence. 

The eligible instructors are coded with a four digit ID 
number. The first three digits are an index and the last is 
a parity check used to catch keypunch errors. The parity 
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check involves matching this digit with the last digit of the 
dot product of the first three digits in the ID with the vector 

( 3 , 5 , 1 ) . 

It is useful to think of the returned ballots as a 
huge data matrix. The rows correspond to the eligible faculty 
and each column represents a ballot. Thus each column contains 
at least five entries with a one opposite the row matching the 
faculty member receiving the first place vote, a two in the 
row representing the second choice, a three in the row repre- 
senting the third choice, and some mark in all positions cor- 
responding to other faculty identified by that ballot. All 
other positions remain void. 

At the right border of this matrix one can tally four 
quantities X^, N for each instructor. For i = 1,2,3 

the value X. is the number of votes of rank i received, and 
1 

N is the number of ballots that identify the particular 
instructor. These four quantities have been used to create 
a score of the form 

S = (w l X l + W 2 X 2 + w 3 X 3 )/N (2.1) 

where w^ , ^ 2 > w 3 are nonnegative weights attached to the 
values of first, second, and third place votes, respectively. 
The early computer programs use the numbers 4, 2, 1 for the 
weights. Also, for each instructor, is recorded the number D 
of ballots that contain a statement supporting his nomination 
for first place. Clearly D <_ X^ for each instructor. 
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At the bottom border of this data matrix, one can tally 
the number K of eligible faculty identified on each ballot. 
These numbers were used for monitoring purposes to see whether 
the voters in the various categories and curricular areas had 
the same distributions for number of instructors identified 
per ballot. Further use will be described later. 

It is useful to comment on the extreme inbalance of 
the data. If all ballots identified the same number of instruc- 
tors and all instructors were identified by the same number 
of ballots then a highly defensible selection procedure could 
be devised. In reality we are a long way from this ideal. 

Some arbitrary choices must be made and the performance of 
the system must be monitored. 
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III. 



EARLY APPROACH TO BALLOT DATA ANALYSIS 



The score S was used to rank the instructors in 
decreasing order. This score did not necessarily determine 
the winner. It was used primarily as a device to order the 
instructors so that the strongest ones appeared together at 
the top of the list. 

The score formula (2.1) carries some presumptions with 
it. Division by N presumes that a good faculty member's 
ability to poll ranking votes, i.e. the , is proportional 
to N the number of ballots that identify him. The value N, 
in turn, is thought to be roughly proportional to the number 
of students he has taught. Thus division by N was viewed 
as an equilizer for the problem of unequal exposure to students 
by faculty. The system of weights represents the worth of 
first, second, and third place votes relative to each other. 

The choice (4,2,1) was made arbitrarily and not without con- 
siderable dissent. Several alternative weighting systems were 
proposed including the set (3,2,1) which is the one offered 
by Condorcet, Laplace, and others in theories of elections 
(see [1,4]) . Note that only the relative proportions of the 
weights are important, not their individual values. Clearly 
the set should satisfy the constraint 

W 1 — w 2 — w 3 — * (3.1) 

The set (3.1) will be called the set of admissible weights. 
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Historically, the first selection committee agreed 
rather quickly that it was appropriate to divide each X i 
by N. It had difficulty in deciding what to do next. At 
this point the data summary consisted of a vector (for each 
instructor) 

x = (x^ x 2 , x^) (3.2) 

in the first octant of three space, where x^ = X^ : N. 

The goal is to choose one of these points as the 'best' 
one in an agreeable and defensible way. Generally there are 
two or three hundred of these points. 

The following screening procedure was applied and 
successfully reduced the number of points under consideration 
to a handful. The idea (borrowed from game theory) is to 
eliminate from consideration all eligible faculty who are 
"dominated." An instructor's score vector x’ is said to 
be dominated if there exists another score vector x" such 
that the projected (one-dimensional) scores using Equation (2.1) 
satisfy 

S' < S" for all admissible weights. (3.3) 

HISTORICAL NOTE ; The original computer codes performed this 
screening in two stages. The first stage searched for vectors 
x" that dominated x' "absolutely." That is 

X 1 - x l' x 2 - x 2' x 3 - x 3 
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with strict inequality in at least one of the three oositions 

The second stage considered only the remaining score 

vectors and eliminated those that were "completely dominated" 

according to (3.3). This is readily accomplished as the set 

of admissible weights (normalized so that w + w + w =1) 

12 3 

is a convex set having extreme points (1,0,0), (1/2, 1/2, 0), 

and (1/3, 1/3, 1/3). Since a convex set is determined by 
its extreme points, and because of the separating hyperplane 
theorem, the inequality (3.3) need only be checked at these 
three points. If S' < S" at all three points then S' 
is completely dominated and removed from further considera- 
ation. The committee can restrict its attention to the un- 
dominated instructors. 

The data from the first year of the award are interest- 
ing. The screening by absolute dominance resulted in thirteen 
eligible teachers from an original list of 249. Further 
screening using complete dominance reduced this to three. 

The three x vectors and corresponding values of 
N appear below. 
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The subscripts of the instructors represent ranks according to 
the initial ordering. If the weights are allowed to "float" 
then each of the three remaining instructors can achieve the 
highest score depending upon how the weights are chosen. The 
choices are presented in Figure 3.1 using barycentric coordi- 
nates ([10]). The coordinate system is the simplex (w^ , W 2 > W 3 ) 
constrained by W 3 + W 2 + = 1 and all _> 0 . The smaller 

right triangle is the set of admissible weights (3.1). The 
three line segments serve to illustrate the weights that pro- 
duce the equality of the pairs of scores S 3 = 83 , S 3 = S 3 , 
and S 2 = S^. The two outside lines partition the right 
triangle into three subsets which determine the decision rule. 
If the weight vector is chosen in the left subset then T 3 
is the winner. It it is chosen in the right subset then T 3 
is the winner, and otherwise T 2 is the winner. It is noted 
that line S 3 = S 3 does not affect the decision partition 

because T. can win only if S. > S. for both values of 
1 1 3 

j / i. For reference purposes a quadrilateral has been drawn 
depicting the region where the relative weights satisfy the 
inequalities 

1/3 <_ W 2 /W 3 <_ 2/3, 1/3 <_ w 3/ w 2 i. 2/3 



The weighting scheme having the ratio 4:2:1 is at the center 
of this quadrilateral. 

By using this screening procedure the committee can 
reduce its choices to a few (in this case, three) and make 
its selection without directly choosing a set of weights. 
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It can also seek further information about the choices. For 
example the quality of the supporting statements can be (and 
has been) examined. 

Returning to the data of the first year of the award, 
a modest "paired comparison" analysis provided the selection 
committee with some additional information about the instruc- 
tors T^, T 2 , an< ^ T 3 * There was only one ballot that iden- 
tified both and T 2 and it made no preference. There 

were 19 ballots that identified in conjunction with 

and 13 other ballots that identified both and T^. These 

"head-to-head" comparisons provided the information that gen- 
erally T 2 was preferred to T^ by the 13 ballots and that 
T^ was preferred to T^ by the 19 ballots. Some additional 
support for T^ was found by including some information 
contained in further paired comparisons among the "top twenty" 
teachers as defined by the score (2.1) using 4:2:1 weights. 

An arbitrary rule was adopted that allowed direct comparisons 
only if at least five ballots identified both members of the 
pair. (One exception was allowed in the case of the three 
ballots that unanimously preferred T^ to T^ . ) The result 
was a set of "strings" ("~" denotes "tied score, non-zero" 
and ">" denotes "is preferred to") 

T 2 > T 15 ' T 3 ' 

T 2 - T 5 > T 3 > T 4 > T X1 > T ir etc. 
with virtually no conflicting preference patterns. 
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This information was made available to the selection committee 



for use as they saw fit. The complete data for these paired 
comparisons is contained in Ref. [7]. The vote of the 
selection committee resulted in the choice of T^. Only at 
this point was the winner's name unveiled. 

The data were studied for other characteristics 
(Ref. [7,2]) and a few are mentioned now. A principal com- 
ponent analysis of the x^, , x^, revealed that the direc- 

tion (in weight space w) of maximum variance is very nearly 
4:2:1. Using these weights, the distribution of positive 
scores is exponential, specifically 

P{S > z | S > 0} = e“ 2z 

The distributions of N (number of ballots that identify 
particular faculty) and K (number of faculty identified 
on the ballots), are both skewed positively. For 1970, 
the means and standard deviation are 



y(N) = 64.5 


0 (N) = 40.4 


p (K) = 15.5 


a (K) = 10.5 



About 50% of the on-board students returned their ballots. 
Chi-square contengency table analyses showed no evidence of 
dependence of score on either voter category, or on the 
instructor's academic rank. 
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IV. EXPERIENCE, TRANSITION AND RECENT STUDIES 

In each of the first four years that this system was 
in use a master's thesis was written [2,3,7,11] which served 
to monitor the process and look for anomalies. A major 
anomaly that was noticed early was that many faculty voters 
tended to identify excessively large numbers of other faculty 
on their ballots. This was viewed as disingenuous and, by 
1975, resulted in the disfranchisement of faculty as voters. 
At about the same time it was decided to invalidate ballots 
which identified too many faculty. Specifically, the voters 
were instructed to identify a number K of eligible faculty 
in the range 

5 <_ K <_ 25 (4.1) 

Also, since the curricular officers usually are also alumni, 
they were removed from the voter category list. 

The selection committees of 1973-74 recognized that 
for each winner there were a substantial number of others 
equally deserving of recognition. They recommended that 
multiple awards be made, but this was rejected by the admini- 
stration. Instead, the custom began of identifying a set of 
ten or so "honorable mentions" and these faculty have always 
been recognized with pay step increases. 

As more data and experience become available, the 
selection committee began to include "past performance" 
data in making their selection. In recent years there has 
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been a very strong tendency to choose the winner from among 
those high scoring faculty who also have a good historical 
record in the honorable mention list. This practice has intro- 
duced some rather unfortunate biases since the record keeping 
process did not keep track of the corresponding historical 
eligibility of faculty. Moreover it was noticed that the 
honorable mention list had a bias favoring faculty with small 
values of N (number of ballots identifying) . Such a bias 
leads to a tendency of favoring those who do not teach many 
students . 

In order to respond to these problems it was necessary 
to perform some exploratory data analysis to complete ballot 
data. Unfortunately no prior year's data had been saved. The 
author undertook this work at the close of the 1978 balloting 
and those data as well as the current 1979 data were studied. 
The important things learned are described next. 

Although the selected winner is not necessarily the 
individual with the largest of the aforementioned scores, S, 
the top ten have been determined by this score and the 
arbitrariness of this score has come under scrutiny. Moreover, 
as already stated, this scoring system tends to favor faculty 
having the smaller values of N. This is illustrated in 
Figure 4.1 which contains scatter plots of score (using 4:2:1 
weights) against N. The top diagram shows the bias favoring 
instructors with small N. The bottom diagram shows how the 
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scatter plot would look if no normalization by N were 
applied. Thus, under the current system, the less well-known 
instructors have the advantage. 

Some studies have been made in an effort to correct this 
situation. First, a modified score of the form 

S' = (w 1 X 1 + w 2 X 2 + w 3 X 3 )/N P for 0 £ p _< 1 (4.1) 

was considered for several values of the exponent p and 
several sets of weights. According to this model a faculty 
member's ability to accumulate ranking votes is an increasing 
concave function of N. The effect is illustrated in Figure 4.2 
which contains scatter plots of S ' vs N p for values of 
p = .75, .5, .25 (and the same 4:2:1 weights). Interestingly, 

it turns out that the exponent p = .75 does a good job of 
making the high scores uncorrelated with N and this result 
appears to hold for a rather large spectrum of weights. Support 
for this point is contained in Appendix A. 

Deeper considerations uncovered the fact that the 
ballots contain more information than was being used. Specifically 
it was noticed that under the present scoring system a first place 
vote (or any other rank for that matter) counts the same regard- 
less of the number of faculty with whom he is compared, i.e. it 
is the same whether that individual is the best of five or the 
best of twenty-five. Clearly the latter condition provides more 
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Scatter Plots of N vs Score for p = .75, .5, 
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information than the former and it is proposed that this be 
reflected in the accounting system. Specifically let us view 
each ballot as a rank distribution. If a ballot identifies a 



total of K instructors, then the top ranked individual on that 
ballot represents the K/(K+l)th quantile of that distribution, 
the second ranked individual represents the (K-l) / (K+l) th 
quantile, and the third the (K-2) / (K+l) th quantile. All others 
must be viewed as tied for positions 4 through K and they are 
awarded the average rank, i.e., the (K-2 ) /2 (K+l) th quantile. 
Using this system each ballot can contribute more information 
to the data summary and to the score of each of its identified 
instructors . 



where the sums are taken over all ballots that identify that 
instructor. With this change, a new weighted scoring system 
is proposed, viz. , 



More specifically, the new information collected for 



oP~ 

each instruction consists of N (as before) and 



Z 1 = l K/ (K+l) , 



Z 2=1 (K-l) / (K+l) 

Z 3 = l (K-2) / (K+l) 
Z 4 = l (K-2) /2 (K+l) 



(4.2) 




(4.3) 
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was studied (using the 1978 data) 



* 

The behavior of the scores S 

for the weighting systems (3:2:1:0), (5:4:3:1), (4:3:2:1), 

(1:1:1:0), and (1:1:1:1) and p = .75, .5 and .25. Figure 4.3 

* P 

contains the scatter plots of S vs for the 5:4:3:1 weights. 

Again p = . 75 produces scores whose large values are uncorre- 
lated with N. Also this result appears to be robust as the 
weighting system is allowed to vary over the above listed systems. 
Appendices A and B contain further data supporting these points. 

Although the scatter plots exhibit shape that does 
not depend on the weights, the system of weights is quite 
important in determining exactly who becomes an "honorable 
mention" and exactly what is placed in the "past performance" 
record of faculty. Since the newly proposed score has these 
arbitrary inputs it is important to monitor its behavior over 
time and adjust the inputs if necessary so that the high scores 
remain uncorrelated with N. Further, since the inputs are 
arbitrary so are the resulting rankings of faculty, and the past 
performance record should not contain either score or rank but 
a general index or grade of performance. Since the system is 
geared to identify good instructors it is recommended that a 
number, say 5%, be designated as class A instructors and the 
next layer, say 15%, be remembered as class B instructors. The 
remaining eligible faculty should be so marked and the values 
of N should be recorded for all. The question of how to 
designate the class A and B instructors is discussed later. 
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Scatter Plots of N vs Proposed Scores for p = .75,. 5,. 25. 

1978 Data 



21 



V. PAIRED COMPARISONS 

The proposed scoring system utilizes more, but not 
all, of the information contained in the ballots. Since each 
ballot makes some direct comparisons between individual faculty 
it should be possible to gather and summarize this direct 
information. The construction of a square win-loss matrix 
is useful to this end. The order of this matrix is the number 
of eligible faculty. The entry in row i and column j is the 
number of ballots that rate instructor i higher than instructor j. 
Ties (i.e. both i and j identified on a ballot but neither 
awarded a ranking position) count one-half each. All diagonal 
(i = j) entries are zero. 

The first step in analyzing a win-loss matrix is to 

identify the subsets of comparable instructors. Thus the list 

of instructors must be partitioned into several sets. If two 

instructors (i,j) appear in the same set then there exists a 

string (possibly empty) of instructor indices (k^, ... , k^) 

such that all consecutive pairs in the string i, k^, k. 2 , . . . , ^ r 'j 

are directly comparable (i.e., either the (k s , ^ s +l^ en try 

in the win-loss matrix or its transpose (k ,k ) or both 

s + 1 s 

are positive) . If two instructors appear in different sets 
then they are not directly comparable themselves, and there 
exists no intermediate string of pairwise directly comparable 
instructors connecting them. In this way the class of instructors 
is partitioned into sets. Within each set there is information 
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that has a direct bearing on the question of ordering the 
instructors in that set. For instructors belonging to distinct 
sets, there is no information for comparing them. 

Scaling methods are available to order linearly all 
instructors within each set. This does not mean that such 
orderings within sets are unique. If the original data are 
highly coherent (i.e. little controversy among the voters in 
forming direct comparisons) then different ordering methods 
will produce essentially the same results. Different methods 
may not produce the same orderings within a subset if there 
is substantial disagreement in the direct comparisons provided 
by the voters. The degree of coherence (or disagreement) within 
an individual subset can be indicated in two ways: 1) Compute 

a 'stress' function that measures the comparative difficulty 
in assigning positions to instructors, and 2) Perform the 
linear ordering in more than one way and compute a measure 
of correlation or concordance of the results. The remainder 
of this section deals with the description of two methods for 
ordering the competitors in a win-loss array and a comparison 
of how they perform when applied to our ballot data. 

Classical Scaling 

If A is a win-loss matrix then the (i,j)th entry of 
T 

N = A + A (A transpose) is the total number of contests 
between i and j. If A is divided by N elementwise, the 
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i,jth element of the resultant, P^j/ is the empirical prob- 
ability that i is preferred to j (according to all those 
who have examined both) . This probability is converted to an 
approximately normal random variable, 0^ , by use of the 
Freeman-Tukey transformation [6]. 

The classical scaling model Fef. [8] assumes that the 
positions of the teachers, T^ , T 2 , • • . are interval scale 
quantities on a normal scale. Thus each 0^ is viewed as 
an estimate (or realization) of T. -T.. The scale values 

1 1 

T^, T 2 , • . • are estimated by means of a weighted least squares, 
i . e . , 

2 



min 



T^ , • • • , j 



l l w . . ( 0 . . -T. + T . ) 



11 11 



1 



(5.1) 



where the weights 



w. . = N. . + .5 
il il 



(5.2) 



are inverse variances of the 
available for each subset of comparable instructors. The 



0^ j . A unique solution is 



partial derivatives with respect to T , when set equal to 
zero, lead to the system of equations 



y w . = y w .(0 . + t.) 

L . r] h rj rj j 



(5.3) 



which can be solved iteratively, initializing all T_. = 0. 

Let {T.} be the solution and define 
1 
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it is a measure of the 



as the stress value for T\ , i.e., 
difficulty in fitting TV into a linear scale containing the 
others. Note that the total stress £ <(k is the originally 
minimized objective function. If it is near zero, then the 
data are well behaved. If it is large then the data are 
being forced, under considerable stress, into a line. The 
quantity 

(l vnw,/ 72 (5.5) 

is a pooled standard deviation of the residuals of the scaling 

/N /\ 

intervals T . - T . . 

i D 

Ford Weights 

The model of Lester Ford, Jr., Ref. [5], assumes the 
existence of a system of weights or "odds" {w^} having the 
property that, P — / the probability that i is preferred to j, 
is given by 

p..=w./(w.+w.) (5.6) 

ID i i D 

Under the assumption that the contests represented by the 
data in the win-loss matrix are independent contests, the 
weights {w^} can be estimated by maximum likelihood. A 
unique solution will exist for each set of comparable instructors 



and it can be found by iterative techniques. The log of the 
likelihood function is 

£nL=^^a..[£nw^-£n(w^+w.)] (5.7) 

i^j 13 1 3 

where {a .} are the elements of the win-loss matrix. The 
iD 

system of partial derivatives of (5.7) with respect to w^, 
leads to the system of equations 

a . = w I N ./(w + w.) (5.8) 

ri r H ri r j 

d 

which can be solved by iteration using the "win percentages" 
of the instructors to initialize the (w^) . The quantities 

<J> . = - £ a . . [£n w - £n(w. + w.)] (5.9) 

1 j ID i ID 

can be used for measuring the stress in estimating w^ . If the 
total stress is large then the likelihood function is flat near 
the maximum. 

Both the Ford weights and the classical scaling models 

were applied to the 1979 data. Due to space limitations in 

the computer, the application was limited to a 60 by 60 win-loss 

matrix consisting of the top 60 ranked instructors of 1979 

* 

ranked according to the newly proposed score S (with p = . 75 
and weights 5:4:3:1). This class of instructors formed a 
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- Weights 


of the Ford Model 








CS 


= Score by 


Classical 


Scaling 







ID = Instructor identification: rank by basic 
1979 scores S* 

Stress = % of total stress. 

TABLE 5.1. Summary of Paired Comparisons Study 
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single set of comparable instructors. The results appear 
in Table 5.1. The two methods agree quite well within them- 
selves (Kendall's coefficient of concordance is 0.98) . The 
total stress is rather large for each (the standard deviation 
in (5.5) is 0.48) . Each of the scalings in Table 5.1 are pre- 
sented in top-to-bottom order. The individual instructors are 
identified by their ID numbers which are their ranks (top-to- 
bottom) using (4.3). See Appendix C for further summaries of 
the 1979 data. 

Although the two scaling methods agree within themselves, 

they are in only modest agreement with the ranking provided by 
* 

the score S . It was decided to vary the inputs (weights 

* 

w and exponent p) of S to see if this condition could 
be improved. The results appear in Table 5.2 which contains 
the values of Kendall's tau. 



Wts p 


1.0 


. 75 


.r 


WO = 3:2: 1:0 


.44/. 45 


.42/. 43 


. 32/. 33 


W1 = 5 : 4 : 3 : 1 


.49/. 49 


.41/. 42 


. 27/. 28 


W2 = 4 : 3 : 2 : 1 


.48/. 48 


.40/. 41 


.26/. 27 


W3 = 1:1: 1:0 


.48/. 49 


.4 3/. 4 4 


.29/. 30 


W4 = 1 : 1 : 1 : 1 


.56/. 56 


.26/. 25 


.06/. 07 



★ 

Kendall's t Computed Between S and FW/CS 



TABLE 5.2. Concordance Coefficients 

19 79 Data 
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Examination of this table shows that the best agreement 

* 

of S with paired comparison scalings occurs at the original 

normalization, p = 1. This in turn suggests that this scaling 

also has a bias favoring instructors with small values of N. 

On the other hand, the changes in instructor position provided 

by the paired comparison scoring provides information that is 

★ 

unavailable in S . When an instructor who is ranked low down 

★ 

according to S appears high on a paired comparison list, we 

* 

know that his low value of S is due to the fact that he has 

been compared w’ith a strong set of instructors on the ballots. 

Similarly when we see an instructor who is ranked high on the 
* 

S list appear low down on a paired comparison list we know 
that he faced weaker competition on the ballots. 

★ 

Since it is difficult to get the advantages of S 
and paired comparisons together in a single scoring system, 
it seems wise to include both in the data summaries to be re- 
viewed by the selection committee. However, the Ford weights 
and the classical scaling are not both needed. The latter is 
selected arbitrarily for use in the proposed data summary 
system. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 

Generally, the award has a good reputation. The system 
has always selected a good teacher for the recipient. The 
practice of choosing as winner an instructor with a good 
"past performance" record in the balloting is a good one, but 
changes are needed to remove biases (with N) when considering 
past performance. The ballot validation restriction of 
5 < K < 25 is arbitrary but wise since it reduces the negative 
effect of a voter listing a large number of instructors. 

Further adjustment for this effect is possible (using (4.2) with 
w^ > 0) and is considered worthwhile. 

In order for the system to maintain its dignity the 
selection committee should monitor the responses in terms of 
watching the distributions of ballots returned by curricular 
area relative to the number sent out, the distributions of 
number of faculty listed on ballots by curricula area, and 
distribution of ballots to alumni by curricula area. Inbalances 
in these distributions should be avoided and investigated 
when they occur as they may signal the presence of unfair 
practices. Computer programs are being written to 
provide data summaries of these distributions. 

Because of the arbitrary inputs, the ranking of instruc- 
tors is arbitrary and it is nonsensical to make firm distinctions 
for the historical record. The balloting system is geared to 
identify good instructors. It is reasonable to select a 
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top 5%" and a "next 15%" for storage--the former being 



marked with an "A" and the latter with a "B" . All other 
eligible faculty should be identified with an "E" and ineligible 
faculty with an "I". The values of N should be kept also. 

A set of IBM cards is being prepared to contain this record. 

Also a computer program is being written to facilitate the 
yearly updating of the record. 

The exponent p and weights W = (w^ :w^ : :w^ ) of 
* 

the new score S are arbitrary inputs. The values p = .75 
and W = (5:4: 3:1) were chosen by the 1978 Selection Committee 
for the following reasons: 

a) The exponent p = . 75 serves to avoid a correlation 
with N of the high scores. 

b) The weights W = (5:4: 3:1) provide equally spaced weights 
for the ranking positions and also allows a positive 
contribution to the score of eligible faculty who are 
listed on a ballot but not ranked. The policy helps to 
offset the manipulative effect of ballots that list a 
large number K of instructors. See Equation (4.3). 



Future Selection Committees should monitor the performance 
of the inputs p and W. Computer programs that produce output 
similar to that in Appendix A are being written for this purpose. 

Finally, the data summaries supplied to the Selection 
Committee will consist of three pages of output: 
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(i) For each of the top 60 instructors (according to the 
* 

score S ) will be listed (see Appendix D) 



(ii) 



* 



Rank by decreasing values of S 
N: the number of ballots that identify him 
D: the number of ballots that provide a 
supporting statement 

* 



S : the score 

X^,X 2 ,X^: the total accumulated 1st, 2nd and 3rd 

place votes 

Z^,Z 2 ,Z^,Z^: the total accumulated quantile counts 

of the four types of responses 
For each string of comparable instructors in the top 
60 will be listed (see Table (5.1)) 



Rank according to localized (and decreasing) CS 

* 

Rank (ID) according to S 

N: the number of ballots that identify him 

CS : score by classical scaling 

Stress: as a percentage of total stress for 

that string. 

(iii) For each of the top 30 instructors will be listed 
(similar to Appendix D) 

* 

Current values of N, S , Rank by CS and 
string identification 
Recorded values of N and grade code 
( A , B , E , I ) for each of the past 
seven years. 



32 



It is expected that the first page will provide the 
committee with adequate numerical information so that its 
members can visualize the entire body of data. This can 
serve as a general basis of comparison. The second page 
will help identify such instructors who have a relatively low 
S* because they found strong competition or who have a 
relatively high S* because they faced weaker competition. 
The past performance data is contained on page 3 together 
with excerpts from the first two pages. The information 
on these three pages provides the basis for designating a 
winner, the top 5% of the eligible instructors, and the 
next 15%. 
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APPENDIX A 



Further study of the distribution of N (number of 
ballots that identify the instructor) as it relates to the 

large scores S (S = (w^Z^ + '^ 2^2 + w 3^3 + w 4^4^ T ' 

To set the standard, the overall distribution of N 

for 1978 is summarized as follows. 

MIN LOQ MEDIAN UPQ MAX SIZE 

1.00 15.50 28.00 42.00 101.00 212.00 

where LOQ and UPQ stand for lower and upper quartiles, 

respectively. The above format is the output of an APL program 

called CONDENSE and is used liberally in what follows. See [9], 

More specifically, a program called DECOMP was written 

★ 

which takes the top 60 scores S (for given weights W and 

power p) , decomposes them into six sets containing 10 instructors 

★ 

each (according to decreasing S ) , and summarizes the corre- 
sponding six sets of 10 values of N. These summaries are 
in the form of CONDENSE and each is followed by a set of vertical 
BOXPLOTS (Ref. [9]). The following seven figures contain the 
results. The choice p = . 75 is supported rather generally as 
a scoring scheme whose high scores favor neither large nor small 
values of N. It also appears that attention should be drawn to 
Fig. A-7, especially in the light of the behavior of the weight 
system W3 = (1:1:1:0) coupled with p = 1 in Table 5.2. 
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FIGURE A-l 

S* = (5Z 1 + 4Z 2 + 3Z 3 + Z 4 ) T N* 5 
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FIGURE A- 2 

* 7 ^ 

S = (5 Z + 4 + 3Z + Z ) t N 
12 3 4 
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FIGURE A- 3 
S* = ( 4 Z 1 + 3Z 2 4- 2Z 3 
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FIGURE A-4 
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(4Z 1 + 3Z 2 + 2Z 3 + Z 4 ) t N* 75 
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39.50 


59. 00 


83.00 




FIGURE A- 5 



S* = (4Z 1 + 3Z 2 + 2Z 3 + Z 4 ) t N 



SIZE 
10.00 
10.no 
10 . 00 
10 . 00 
10.00 
10 . 00 
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io decoup n 



MIN 


LOQ 


36, 00 


48.00 


13.00 


18.00 


6. 00 


25.00 


20.00 


30.00 


11 . 00 


23.00 


24. 00 


27.00 



MEDIAN 


UPQ 


60 . 50 


71. 00 


31 . 50 


66 . 00 


42.00 


59. 00 


33.50 


59. 00 


39. 00 


45.00 


37. 50 


46 . 00 



MAX 


SIZE 


101 . 00 


10.00 


83.00 


10.00 


66 . 00 


10 . 00 


61. 00 


10. 00 


50 . 00 


10.00 


52. 00 


10.00 





i 



T 




x 



FIGURE A- 6 



* 

S 



(Z x + Z 2 + Z ) i N- 5 



40 



10 DECOUP n 



MIN 


LOQ 




UEDIAN 


6. 00 


10.00 




23.00 


2. 00 


20 . 00 




30 . 00 


7.00 


25:oo 




35.50 


3. 00 


24 . 00 




40 . 00 


16 . 00 


27 . 00 




35.00 


11.00 


21 . 00 




41.50 






X 








X 








X 






o 


X 


X 






X 


X 






X 


X 




X 


1 


1 


X 


X 


1 1 


1 1 


X 


X 


1 1 


1 1 


_i_ 

i i T 


_±_ 
1 1 


1 1 
1 1 


1 1 
1 1 


i i i 

1 1 1 


1 1 
1 1 


1 1 
1 * 1 


1 1 
! 1 




1*1 


1 1 


1 * 1 


1 * 1 


1 1 


1 1 


“t“ 


1 1 1 


T" 


T _ 


X 


T~ 


X 


X 




X 


X 


X 




X 




X 





UPQ 


MAX 


SIZE 


48.00 


71 . 00 


10.00 


48.00 


63.00 


10.00 


50.00 


78 . 00 


10.00 


63.00 


101.00 


10.00 


66.00 


83.00 


10.00 


4 5.00 


50 . 00 


10.00 



T = T 



T 

X 



FIGURE A- 7 



S* = (Z ] _ + Z 2 + Z 3 ) t N 
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APPENDIX B 



EFFECT OF VARYING INPUTS 



The 1978 data were used to explore the effect of 



varying the exponent p and 
Specifically, 

P 

and 

WO 

W1 

W2 

W3 

W4 



the weighting system W. 

.75, .5 

: 3 : 2 : 1 : 0 
= 5 : 4 : 3 : 1 
: 4: 3:2:1 
: 1 : 1 : 1 : 0 
: 1 : 1 : 1 : 1 



The index numbers in Table B-l are the ranks according to the 
original score S with p = 1 and W = 4:2:1. Table B-2 
contains the corresponding values of N. 
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p 



75 



5 



P- . 



Rank 


WO 


W1 


W2 


W3 


W4 


WO 


wi 


W2 


W3 


W4 


1 


4 


4 


4 


4 


4 


4 


4 


4 


4 


20 


2 


2 


2 


2 


5 


10 


10 


10 


10 


10 


4 


3 


1 


* 5 


5 


10 


7 


5 


7 


18 


5 


10 


4 


5 


10 


10 


2 


5 


7 


5 


7 


7 


18 


5 


3 


1 


1 


1 


18 


11 


18 


5 


18 


32 


6 


10 


7 


7 


7 


21 


18 


11 


11 


11 


21 


7 


7 


3 


3 


3 


28 


28 


28 


28 


21 


7 


8 


11 


11 


11 


11 


11 


2 


21 


21 


28 


11 


9 


m 


18 


10 


8 


32 


15 


15 


15 


32 


36 


10 


15 


15 


15 


13 


15 


21 


32 


* 32 


15 


60 


11 


8 


21 


21 


18 


2 


3 


24 


24 


8 


24 


12 


12 


8 


8 


15 


36 


32 


2 


36 


2 


31 


13 


16 


16 


12 


21 


24 


24 


36 


2 


33 


15 


14 


21 


12 


28 


16 


31 


12 


3 


31 


16 


5 


15 


9 


28 


16 


19 


33 


16 


31 


3 


3 


48 


16 


2 R 


13 


32 


32 


8 


8 


16 


16 


24 


33 


17 


13 


32 


9 


33 


16 


1 


12 


12 


13 


57 


10 


6 


9 


24 


9 


3 


36 


8 


8 


19 


55 


19 


14 


24 


13 


28 


12 


31 


33 


33 


36 


37 


20 


24 


19 


14 


12 


60 


27 


60 


60 


60 


71 


21 


19 


14 


19 


30 


37 


33 


27 


27 


31 


76 


22 


32 


33 


31 


14 


23 


9 


48 


48 


27 


61 


23 


31 


31 


33 


24 


48 


19 


19 


23 


12 


16 


24 


33 


3 6 


36 


27 


13 


23 


23 


37 


55 


27 


25 


23 


23 


2 3 


6 


57 


14 


37 


19 


30 


79 


26 


27 


6 


6 


34 


19 


13 


1 


1 


1 


126 


27 


36 


27 


27 


23 


55 


48 


9 


9 


37 


23 


28 


20 


30 


37 


31 


34 


60 


13 


57 


9 


12 


29 


30 


37 


30 


37 


30 


37 


55 


5 5 


23 


85 


30 


37 


34 


34 


26 


1 


6 


14 


14 


4 R 


127 


31 


26 


48 


48 


30 


27 


26 


57 


13 


34 


135 


32 


2 5 


60 


60 


55 


14 


34 


34 


3 4 


14 


53 


33 


34 


2 6 


20 


60 


79 


2 5 


30 


53 


26 


0 


34 


22 


20 


26 


40 


9 


30 


61 


61 


61 * 


91 


35 


4 R 


55 


2 5 


48 


61 


57 


26 


71 


40 


141 


36 


35 


40 


22 


70 


76 


55 


53 


3 0 


70 


107 


37 


40 


2 5 


35 


20 


S3 


20 


71 


45 


76 


123 


38 


39 


35 


39 


4 2 


71 


35 


76 


2 6 


57 


115 


39 


60 


22 


40 


35 


85 


53 


44 


44 


71 


34 


40 


42 


42 


57 


61 


4 5 


22 


45 


76 


35 


45 


41 


53 


39 


53 


25 


44 


45 


25 


25 


53 


118 


42 


29 


53 


55 


39 


67 


44 


35 


35 


79 


44 


43 


44 


57 


4 2 


22 


39 


61 


79 


79 


44 


159 


44 


45 


44 


44 


67 


40 


39 


56 


56 


6 


129 


45 


57 


61 


4 5 


53 


91 


40 


40 


20 


85 


19 


46 


55 


4 5 


61 


52 


77 


71 


20 


39 


67 


77 


47 


52 


52 


56 


44 


70 


56 


39 


22 


42 


67 


48 


56 


56 


52 


77 


42 


42 


85 


85 


77 


56 


49 


61 


71 


71 


76 


107 


52 


6 


40 


20 


104 


50 


54 


70 


29 


79 


56 


76 


22 


6 


2 5 


82 


51 


71 


76 


76 


57 


127 


54 


52 


5? 


52 


101 


52 


41 


67 


79 


41 


35 


79 


6 7 


67 


56 


145 


53 


58 


79 


67 


85 


52 


29 


42 


42 


39 


3 


54 


67 


29 


54 


71 


104 


67 


70 


54 


22 


30 


55 


38 


85 


70 


56 


126 


58 


77 


70 


4 5 


70 


56 


76 


54 


85 


69 


118 • 


63 


54 


77 


104 


63 


57 


70 


77 


se 


29 


54 


es 


91 


6 3 


69 


2 


58 


47 


58 


77 


45 


73 


70 


63 


91 


02 


• 54 


59 


17 


6 3 


63 


9 4 


25 


77 


82 


82 


94 


09 


CO 


51 


73 


51 


81 


26 


51 


73 


73 


91 


73 



TABLE B-l. Index Numbers for Several Scoring Systems 
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* 




P * • 


75 






P 


- • 5 






Rank 


wo 


wi 


W2 


W3 


W4 


WO 


Wl 


W2 


W3 


W4 


1 


58 


50 


58 


58 


50 


50 


58 


50 


5ft 


101 


2 


13 


13 


13 


36 


71 


71 


71 


71 


71 


58 


3 


6 


36 


3 C 


71 


48 


36 


40 


6ft 


36 


71 


4 


36 


71 


71 


13 


36 


48 


36 


4ft 


4ft 


6ft 


5 


18 


6 


6 


6 


68 


57 


6 8 


36 


6 0 


7ft 


6 


71 


48 


40 


48 


63 


60 


57 


57 


57 


63 


7 


48 


18 


18 


10 


101 


101 


101 


101 


6 3 


48 


a 


57 


57 


57 


57 


57 


13 


63 


63 


101 


57 


9 


68 


68 


68 


26 


78 


4ft 


4ft 


4ft 


78 


1 6 


10 


48 


48 


4 8 


18 


48 


63 


78 


7ft 


48 


03 


n 


26 


63 


63 


68 


13 


10 


66 


66 


26 


66 


12 


32 


26 


26 


40 


76 


70 


13 


76 


13 


63 


13 


35 


35 


32 


63 


66 


66 


76 


13 


U5 


4 0 


14 


63 


32 


101 


35 


63 


32 


1 ft 


63 


35 


36 


15 


20 


101 


35 


28 


45 


35 


63 


18 


1ft 


66 


16 


101 


18 


78 


78 


26 


26 


35 


35 


66 


45 


17 


18 


78 


20 


45 


35 


6 


32 


32 


18 


61 


18 


11 


20 


66 


20 


18 


76 


26 


26 


28 


59 


19 


20 


66 


18 


101 


32 


63 


45 


45 


76 


46 


20 


66 


28 


20 


32 


83 


50 


83 


83 


83 


61 


21 


28 


20 


28 


25 


46 


45 


50 


50 


6 3 


59 


22 


78 


45 


63 


20 


38 


20 


66 


66 


50 


53 


23 


63 


63 


45 


66 


66 


28 


28 


38 


32 


35 


24 


45 


76 


76 


50 


18 


38 


38 


46 


59 


50 


25 


38 


38 


38 


11 


61 


20 


46 


28 


25 


50 


26 


50 


11 


11 


33 


28 


18 


6 


6 


6 


75 


27 


76 


50 


50 


38 


59 


66 


20 


20 


46 


38 


28 


22 


25 


46 


63 


33 


83 


18 


61 


20 


32 


29 


25 


46 


25 


46 


25 


46 


59 


59 


38 


49 


30 


46 


33 


33 


32 


6 


11 


20 


20 


66 


64 


31 


32 


66 


66 


76 


50 


32 


61 


10 


33 


74 


32 


30 


83 


03 


59 


20 


33 


33 


33 


20 


45 


33 


33 


32 


22 


03 


50 


30 


25 


45 


32 


26 


34 


24 


22 


32 


25 


20 


25 


53 


53 


53 


52 


35 


66 


59 


30 


66 


53 


61 


32 


61 


25 


73 


36 


30 


25 


24 


34 


59 


59 


45 


2 5 


34 


52 


37 


25 


30 


30 


22 


45 


22 


61 


43 


59 


70 


38 


27 


30 


27 


23 


61 


30 


59 


32 


61 


65 


39 


83 


24 


25 


30 


49 


45 


42 


42 


61 


33 


40 


23 


23 


61 


53 


43 


24 


43 


59 


30 


43 


41 


45 


2 7 


45 


30 


42 


43 


30 


30 


45 


56 


42 


16 


4 5 


59 


27 


37 


42 


30 


30 


50 


42 


43 


42 


61 


23 


24 


27 


53 


50 


50 


42 


59 


44 


43 


42 


42 


37 


25 


27 


42 


42 


11 


65 


45 


61 


53 


43 


4 5 


52 


2 5 


25 


22 


49 


28 


4 6 


59 


43 


53 


33 


41 


61 


22 


27 


37 


41 


47 


33 


‘33 


42 


42 


34 


42 


27 


24 


23 


37 


48 


42 


42 


33 


41 


23 


23 


4 9 


4 9 


41 


42 


49 


53 


61 


61 


59 


52 


33 


1 1 


2 5 


22 


4 6 


50 


38 


34 


16 


50 


42 


59 


2 4 


11 


30 


47 


51 


61 


59 


59 


61 


64 


38 


33 


33 


33 


51 


52 


7 


37 


50 


7 


30 


50 


37 


37 


42 


50 


53 


31 


50 


37 


49 


33 


16 


23 


2 3 


27 


10 


54 


37 


16 


38 


61 


46 


37 


34 


30 


24 


25 


55 


11 


49 


34 


42 


75 


31 


41 


34 


4 3 


34 


SC 


59 


30 


49 


25 


56. 


41 


30 


41 


4 6 


41 


57 


34 


41 


31 


16 


39 


49 


52 


41 


25 


13 


58 


17 


31 


41 


43 


37 


34 


41 


52 


47 


38 


59 


3 


41 


41 


30 


30 


41 


4 7 


47 


30 


41 


60 


27 


37 


27 


2 


32 


27 


37 


37 


52 


37 


TABLE 


B-2 . 


Values 


of N 


Corresponding to 


Table 


B-l. 
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APPENDIX C 



DATA SUMMARY OF TOP SIXTY SCORES BY CURRENT METHOD 
AND BY THE PROPOSED NEW METHOD 
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TABLE C-l. 



Rank 


s 


D 


N 


XI 


X2 


X3 


1 


3 • 33 


3 * 


G 


4 


2 


0 


2 


2,69 


6 


13 


8 


1 


1 


3 


2.22 


6 


18 


' 8 


4 


0 


4 


2.00 


17 


58 


24 


9 


2 


5 


1 . 92 


9 


36 


12 


8 


5 


6 


1.73 


4 


11 


4 


1 


1 


7 


1. 50 


o 


4 6 


14 


7 


6 


8 


1 . 58 


5 


26 


7 


4 


5 


0 


1. 55 


4 


20 


5 


5 


1 


10 


1 . 54 


12 


71 


18 


15 


7 


11 


1. 51 


9 


57 


16 


9 


4 


12 


1 . 50 


6 


32 


11 


1 


2 


13 


J. 50 


2 


18 


3 


5 


5 


14 


1.40 


4 


20 


5 


3 


2 


15 


1 . 38 


10 


48 


13 


4 


6 


16 


1 . 37 


4 


35 


7 


9 


2 


17 


1 .33 


1 


3 


1 


0 


0 


18 


1. 32 


4 


68 


16 


11 


4 


19 


1.25 


5 


28 


5 


5 


5 


20 


1 . 14 


4 


22 


5 


2 


1 


21 


1 .13 


6 


63 


10 


12 


7 


22 


1.08 


4 


24 


5 


3 


0 


23 


1.05 


5 


38 


7 


5 


2 


24 


1 . 05 


6 


66 


13 


7 


3 


25 


1 . 03 


5 


30 


7 


1 


1 


26 


1 . 03 


2 


32 


5 


4 


5 


27 


1 . 02 


6 


50 


8 


7 


5' 


28 


1 .01 


13 


101 


18 


10 


10 


29 . 


1 . 00 


2 


16 


3 


2 


0 


30 


1 . 00 


2 


25 


2 


6 


5 


31 


0.97 


7 


63 


11 


7 


3 


32 


0.96 


4 


78 


9 


16 


7 


33 


0.96 


4 


45 


4 


11 


5 


34 


0 . 94 


3 


33 


4 


5 


5 


35 


0.93 


3 


30 


5 


3- 


2 


36 


0 . 92 


10 


76 


15 


2 


6 


37 


0.91 


5 


46 


6 


6 


6 


38 


0.91 


2 * 


11 


2 


1 


0 


39 


0.89 


4 


27 


5 


1 


2 


40 


0.88 


0 


25 


3 


3 


4 


41 


0 .06 


0 


7 


0 


3 


0 


42 


0 . 83 


1 


23 


2 


5 


1 


43 


0 .80 


1 


5 


1 


0 


0 


44 


0. 79 


3 


42 


6 


3 


3 


45 


0 . 77 


4 


43 


7 


2 


1 


46 


0 . 76 


1 


17 


3 


0 


1 


47 


0 . 76 


2 


17 


3 


0 


1 


48 


0 . 76 


5 


66 


6 


12 


2 


49 


0 . 75 


1 


8 


1 


1 


0 


50 


0.75 


3 


16 


3 


0 


0 


51 


0 . 74 


3 


27 


5 


0 


0 


52 


0 . 73 


2 


33 


3 


5 


2 


53 


0 . 71 


4 


45 


4 


8 


0 


54 


0 . 71 


5 


38 


6 


1 


1 


55 


0 .69 


4 


59 


6 


. 2 


13 


56 


0.69 


3 


42 


5 


3 


3 


57 


0 .69 


3 


61 


8 


3 


4 


58 


0 .68 


2 


31 


4 


2 


1 


59 


0.67 


4 


24 


4 


0 


0 


60 


0 . 66 


4 


83 


6 


U 


9 


Score 


Summary 


1978 




(Old System 


W = - 



p = l) 



46 



N 

5 fl 

13 

30 

71 

C 

4 6 

16 

57 

68 

48 

G 3 

26 

35 

32 

L 0 1 

18 

78 

20 

66 

28 

20 

45 

63 

76 

38 

11 

50 

25 

46 

33 

66 

83 

32 

22 

59 

25 

30 

30 

24 

23 

27 

45 

61 

42 

53 

4 3 

33 

42 

61 

34 

59 

37 

5C 

16 

49 

38 



X3 

2 

1 

5 

7 

0 

6 

0 

4 

4 

6 

7 

5 

2 

2 

10 

5 

7 

1 

3 

5 

2 

5 

3 

6 

2 

1 

5 

5 

6 

5 

2 

9 

5 

1 

13 

4 

1 

2 

0 

1 

2 

0 

4 

3 

5 

1 

2 

3 

4 

5 

5 

4 

3 

0 

5 

1 

5 

1 

1 

2 



* 



D 


S 


Z1 


17 


7. 10 


21.61 


G 


0.56 


7.G0 


9 


C .49 


10.96 


12 


G . 34 


1C . 30 


3 


C . 2 6 


.3.03 


9 


G . OG 


12.89 


0 


5 . P 6 


7.11 


9 


5.6 6 


14.22 


4 


5.52 


14.4 7 


10 


5 .32 


12.02 


G 


5.14 


6.93 


5 


5.13 


6 . 32 


4 


4.88 


6.23 


6 


4.87 


10.16 


13 


4.83 


16.16 


2 


4.71 


2 . 58 


4 


4.67 


8 . 08 


4 


4 . 59 


4 . 30 


6 


4.49 


11.68 


5 


4 .46 


4 . 52 


4 


4 .41 


4.60 


4 


4 . 38 


3.67 


7 


4.23 


9.97 


10 


4.15 


13.62 


5 


4 .12 


6.27 


4 


4.10 


3.67 


6 


4.05 


7.16 


2 


3.97 


1 .77 


5 


3 . 86 


5 . 34 


3 


3.61 


3. 55 


5 


3.66 


5 .42 


4 


3.62 


5 . 51 


2 


3.61 


4 . 34 


4 


3.55 


4 . 54 


4 


3 . u 5 


5 . 56 


0 


3.45 


2.81 


5 


3.44 


6.39 


3 


3.4 1 


4.60 


4 


3 . 36 


4.55 


1 


3 . 33 


1 . 89 


4 


3 . 32 


4.64 


4 


3.29 


3.75 


3 


3 .28 


7. 3P 


3 


3.23 


5.27 


1 


3.21 


9.78 


4 


3.16 


6 . a 3 


2 


.3 . 07 


2 .70 


3 


3 . 04 


*..63 


2 


3 .01 


3.59 


0 


3 .01 


0 . no 


1 


2 .90 


2 .62 


1 


2.97 


1.91 


1 


2 .96 


.0.95 


o 


2 .96 


2.73 


0 


2 . 86 


1 . 88 


5 


2 . 80 


5.18 


0 


2 .79 


0.91 


2 


2 .70 


3.66 


0 


2.65 


3 . 50 


1 


2 .62 


2 . 78 


2 . 


Data 


Summary 



22 Z3 



7. 2C 


1.63 


0.90 


0.70 


6.70 


3. 32 
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LEGEND: A = top 5%, B = next 15%, E = Eligible, I = Ineligible. 
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